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I. 


INTRODUCTION 


Radar  guided  missile  simulation  is  one  of  several  key  methods  employed 
to  gain  as  thorough  an  understanding  as  possible  of  radar  missile  performance 
within  an  operational  environment.  Simulation  is  valuable  whether  in  the 
design,  development,  evaluation,  or  deployment  stages  of  a  missile  life  cycle 
because  it  permits  conducting  a  larger  number  of  carefully  designed  trials 
in  a  controlled  environment  than  is  possible  in  flight  tests.  For  the  results 
of  simulation  to  be  a  meaningful  representation  of  actual  missile  performance, 
however,  it  is  essential  that  the  models  of  the  system  and  its  physical  environ¬ 
ment,  which  form  the  building  blocks  or  simulation,  have  a  demonstrable  corres¬ 
pondence  to  reality.  Among  the  models  are  ones  characterizing  the  RF  environ¬ 
ment  which  stimulates  the  missile  seeker.  Verification  and  validation  (V&V) 
of  the  underlying  models  in  a  formal  program  is  therefore  a  necessary  ingredient 
in  a  successful  missile  simulation  effort. 

The  Radio  Frequency  Simulation  System  (RFSS)  program  for  V&V  of  the  RF 
Environmental  Models  utilized  in  hardware-in-the-loop  (HWIL)  radar  guided 
missile  simulation  at  the  MI COM  Advanced  Simulation  Center  (ASC)  is  the 
subject  of  this  document.  The  RFSS,  the  Electro-optical  Simulation  System 
(EOSS),  and  the  Infrared  Simulation  System  (IRSS)  comprise  the  three  physical 
effects  simulators  that  surround  a  hybrid  computing  complex.  The  RFSS  is 
unique  in  its  capability  of  presenting  the  RF  environment  to  the  device  under 
test.  Included  in  its  capabilities  are  the  following: 

o  2  to  18  GHz  frequency  coverage, 
o  polarization  diversity, 
o  four  simultaneous,  independent  targets. 

o  42°  instantaneous  field-of-view  presented  to  a  device  under  test, 
o  real-time  range  delay  for  coherent  signal  processors, 
o  tapped  delay  lines  aud  modulators  for  RF  model  implementation. 

These  capabilities  allow  the  simulation  (in  real-time)  of  any  RF  guidance 
mode,  including  coherent  or  noncoherent  active,  semi-active,  passive,  command, 
beam-rider,  and  track-via-missile  (TVM).  The  reader  is  presumed  to  be 
familiar  with  RFSS  simulation  operations  and  capabilities  as  described  in  the 
RFSS  Users  Guide  [l]  and  the  RFSS  Capabilities  Summary  [2]. 
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The  RF  Environmental  Models  form  one  of  the  baste  building  blocks  of 
the  overall  ASC  guided  missile  simulation.  RF  environment  refers  to  the  com¬ 
posite  of  electromagnetic  signals,  which  stimulate  the  missile  antenna  during 
flight,  and  includes  target  scattering  returns,  clutter,  multipath,  and  ECM 
signals.  During  a  simulation  in  the  RFSS,  RF  signals  are  generated  in  real 
time  and  transmitted  across  the  anechoic  chamber  to  the  seeker  mounted  on  a 
three-axis  flight  table.  To  properly  exercise  the  missile  seeker  hardware 
during  simulation,  the  signals  presented  to  the  seeker  by  the  Environmental 
Models  ideally  should  be  indistinguishable  from  those  encountered  during  an 
actual  combat  flight.  Because  of  incomplete  knowledge  of  the  physical  pheno¬ 
mena  involved  and  constraints  in  time  and  budgets,  it  is  possible  to  achieve 
this  goal  with  only  limited  success.  Within  such  limits,  the  RFSS  Environ¬ 
mental  Model  V&V  Program  is  designed  to  assure  that  the  signals  presented  to 
the  seeker  are  as  realistic  as  possible  and,  furthermore,  to  quantify  this 
degree  of  realism. 

V&V  has  become  an  integral  part  of  the  modern  approach  to  the  develop¬ 
ment  of  large  software-based  systems.  Although  the  specific  objectives  and 
requirements  of  RF  environmental  modeling  differ  significantly  from  those  of 
software  system  development,  the  RFSS  approach  to  V&V  utilizes  the  same  well- 
planned,  time-phased  approach  to  V&V  which  has  been  proven  successful  by  soft¬ 
ware  engineers. 

The  first  stage  of  Environmental  Model  V&V  is  verification  of  the  correct 
implementation  of  the  models.  This  is  accomplished  in  three  stages: 

o  Model  Design  Requirement  Review, 
o  Model  Implementation  Verification, 
o  Periodic  Calibration/Diagnostic  Maintenance. 

Design  review  checks  that  the  mathematical  models  developed  for  a  particular 
simulation  meet  the  requirements  agreed  upon  by  both  the  simulation  customer 
and  RFSS  System  Leader  in  planning  the  simulation.  Verification  measurements 
assure  that  this  mathematical  model  has  been  correctly  implemented  in  real¬ 
time  hardware/software  in  the  RFSS.  Calibration/diagnostic  maintenance  is  a 
routine  series  of  tests  made  on  a  daily  or  weekly  basis  during  simulation 
operations  to  check  that  the  models  continue  to  function  properly.  If  verifi¬ 
cation  standards  are  not  met  at  any  stage  in  this  process,  corrective  action 
is  taken  before  proceeding  to  the  next  step. 


swv  ::r" j. 
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Environmental  Model  validation  addresses  the  much  broader  issue  of 
model  realism.  The  central  issues  are  how  adequately  do  the  correctly 
implemented  models  emulate  the  real-world  environment  and  what  are  the  model 
limitations.  These  issues  are  addressed  by  comparing  simulation  predictions 
with  actual  field  data  or  other  independent  results.  The  extent  and  nature 
of  these  comparisons  are  highly  program  specific.  Depending  upon  the  quantity 
and  type  of  data  provided  to  the  RFSS  by  the  simulation  customer  for  valida¬ 
tion,  various  quantitative  measures  of  agreement  between  simulation  predictions 
and  reality  can  be  computed  and  levels  of  confidence  in  the  simulation  results 
establ i shed. 

As  applied  to  RF  Environmental  Models  utilized  in  HWIL  simulation  in  the 
RFSS,  the  following  definitions  are  appropriate: 

o  Verification--the  process  involving  acquisition  and  analysis  of 

RFSS  measurement  data  which  assures  RF  Environmental 
Models  implemented  in  the  RFSS  meet  their  design 
objectives. 

o  Val idation--the  assessment  and  quantification  of  the  degree  to 

which  RF  Environmental  Models  are  adequate  representa¬ 
tions  of  physical  reality. 

The  general  topic  of  V&V  is  discussed  In  Section  II,  both  as  it  applies 
to  guided  missile  simulation  and  as  it  has  evolved  from  earlier  work  in  soft¬ 
ware  systems  development.  The  specific  RFSS  approach  to  VAV  of  RF  Environmental 
Models  is  delineated  in  Sections  III  and  IV,  respectively. 

II.  V&V  FRAMEWORK 

A  primary  objective  of  HWIL  seeker  simulation  is  to  contribute  to  the 
development  and  assessment  of  weapon  system  performance.  The  aggregate 
expense  of  missile  flight  tests,  including  costs  for  fabrication  of  both 
developmental  missile  hardware  and  target  drones  and  costs  for  operations  of 
the  test  ranges,  makes  more  than  a  limited  number  of  flight  tests  economically 
infeasible.  HWIL  simulation  can  be  used  to  maximize  the  information  obtained 
from  these  tests  by  predicting  through  preflight  simulation  those  scenarios 
critical  to  missile  performance  envelope  assessment  and  by  contributing  through 
post-flight  simulation  to  the  analysis  and  understanding  of  flight  test  results. 


It  is  also  important  that  HWIL  simulation  be  used  to  gain  an  understanding 
of  missile  system  performance  in  those  situations  where  avoidance  of  flight 
test  cost  is  not  the  motivation;  for  example,  where  it  is  required  to  estimate 
performance  versus  projected  threat  capability,  where  quick  reaction  perfor¬ 
mance  assessments  are  needed,  or  where  special  security  requirements  preclude 
tests  susceptible  to  monitoring  by  a  potential  adversary.  Simulation  supports 
systems  analysis,  permits  examination  of  "what  if"  issues,  and  contributes  to 
the  lowering  of  design  risks.  Results  help  to  increase  confidence  in  system 
performance  and  generally  provide  a  data  base  for  proper  management  decision- 
makinq. 

To  achieve  these  and  other  simulation  benefits,  it  is  mandatory  that 
the  realism  of  HWIL  simulation  be  demonstrated,  so  that  a  high  degree  of  con¬ 
fidence  can  be  placed  in  simulation  predictions  and  results.  The  V&V  process 
of  both  the  overall  simulation  and  the  component  simulation  modules  provides 
an  orderly  framework  for  evaluating  the  degree  of  simulation  realism  and 
determining  the  level  of  confidence  which  is  appropriate  to  the  results. 


A.  V&V  Hierarchy 

As  illustrated  in  Figure  1,  a  number  of  factors  are  involved  in 
predicting  the  performance  of  a  candidate  weapon  system.  The  ultimate  measure 
of  realism  is  performance  in  a  real-world  combat  situation.  Fliqht  testing 
closelv  emulates  reality,  although  even  in  this  case  a  surrogate  target  is 
typically  substituted  for  an  actual  threat  vehicle.  Target  realism  vis-a-vis 
the  threat  is  an  issue  which  must  be  satisfactorily  demonstrated  before  flight 
test  predictions  can  be  accepted  with  great  confidence. 
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Figure  1.  Hierarchy  of  performance  evaluation  methods. 
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In  the  remaining  evaluation  methods — HWIL  simulation,  captive  flight 
testinq,  diqital  simulation,  and  design  analysis— certain  aspects  of  reality 
are  replaced  b.v  models  to  gain  scientific  control  over  engagement  scenarios 
and  the  large  number  of  variables  which  can  affect  missile  performance.  The 
degree  to  which  each  method  contributes  to  an  understanding  of  overall 
missile  performance  is  dependent  upon  the  validity  of  these  models.  For  an 
RF  guided  missile,  a  division  can  be  made  into  three  major  elements  which 
contribute  to  missile  performance.  As  depicted  in  Figure  2,  these  are  the  RF 
environment,  which  stimulates  the  seekers;  the  sensor/signal  processor,  which 
processes  the  seeker's  outputs;  and  the  missile  flight  dynamics  system.  RF 
environment  refers  to  the  composite  of  electromagnetic  signals  that  is  avail¬ 
able  to  excite  the  missile  antenna  and  includes  target  return,  clutter,  multi- 
path,  and  ECM  signals.  The  sensor/signal  processor  includes  both  the  missile 
microwave  receiver  and  the  target  acquisition  and  tracking  logic  circuitry 
and/or  computer.  Finally,  the  flight  dynamics  system  includes  the  remainder 
of  the  missile  system  (autopilot,  actuators,  etc.)  and  the  aerodynamic  coupling 
tc  the  surrounding  atmosphere. 


Figure  2.  Major  missile  performance  elements, 


The  tasks  of  V&V  apply  not  only  to  overall  missile  performance  but  also 
to  each  of  these  elements.  Moreover,  in  a  missile  simulation,  it  is  impossible 
to  establish  with  a  high  degree  of  confidence  the  credibility  of  simulation 
results  without  first  verifying  the  correct  implementation  of  the  individual 
models  and  validating  the  correspondence  of  model  predictions  to  reality. 
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The  specific  application  of  these  concepts  to  RFSS  HWIL  simulation  is 
discussed  in  paragraph  II. B. 

B.  V&V  Applied  to  HWIL  Simulation 

The  motivating  factor  behind  HWIL  simulation  is  that  an  RF  sensor/ 
signal  Drocessor  performs  highly  complex  functions  which  are  extremely  diffi¬ 
cult  to  model  realistically  on  a  computer.  By  inserting  actual  hardware  into 
the  simulation,  the  performance  of  this  subsystem  is  by  definition  valid.  The 
imoact  of  this  approach  is  that  the  environmental  and  flight  dynamics  models 
must  not  only  be  modeled  realistically,  but  they  must  couple  with  the  seeker 
and  must  execute  in  real  time. 

In  an  all-digital  simulation,  various  levels  of  detail  may  be 
appropriate  for  the  seeker  model,  depending  upon  the  objectives  of  the  simu¬ 
lation.  To  determine  miss-distance  statistics  or  false  alarm  probabilities 
via  Monte  Carlo  techniques,  a  simplified  model  of  seeker  performance  which 
executes  in  real-time  is  often  appropriate.  Detailed  parametric  studies  of 
seeker  performance  versus  variations  in  the  RF  environment,  however,  require 
complex  seeker  models  which  typically  execute  hundreds  or  even  thousands  of 
times  slower  than  real-time.  It  is  obvious  that  the  V&V  criteria  applied  to 
each  of  these  extremes  must  reflect  the  different  simulation  objectives. 

Similarly,  different  levels  of  detail  are  appropriate  for  the  RF 
Environmental  Models,  depending  on  simulation  objectives.  To  determine  acquisi 
tion  range,  for  example,  a  simple  point  target  may  suffice;  but  to  determine 
end-game  seeker  performance,  a  realistic  extended  target  model  is  reouired. 

In  either  case,  the  validity  of  the  overall  simulation  is  contingent  upon 
utilizing  RF  Environmental  Models,  Seeker  Models  (or  actual  seeker  hardware), 
and  Flight  Dynamics  Models  which  have  been  both  verified  and  validated  over 
the  ranqe  of  parameters  covered  in  the  simulation. 

1.  Flight  Dynamics  Models 

The  missile  dynamics  models  utilized  by  the  RFSS  in  HWIL  simu¬ 
lations  are  developed  and  implemented  by  the  MICOM  ASC.  Verification  of  these 
models  is  of  great  significance;  for  example,  an  error  in  the  model  output, 
which  drives  the  three-axis  flight  table  on  which  the  actual  seeker  is  mounted, 
can  result  in  actual  physical  damage  to  the  seeker.  These  models  are  imple¬ 
mented  through  a  step  by  step  integration  of  previously  verified  submodules 


until  a  completely  verified  flight  dynamics  model  is  assembled.  This  process 
which  uses  a  previously  verified  all-digital  flight  dynamics  model  as  the 
verification  baseline,  is  described  in  detail  in  the  RFSS  User's  Guide  [l]. 

The  Flight  Dynamics  Model  is  validated  by  comparing  a  time  history  of  deflection 
commands  to  the  missile  control  surfaces  generated  by  the  model  with  corres¬ 
ponding  signals  telemetered  from  actual  flight  tests  or  generated  by  a  pre¬ 
viously  validated  simulation  model.  Individual  subsystems,  such  as  the  auto¬ 
pilot,  can  be  independently  validated  if  snecial  provisions  are  made  to  include 
the  appropriate  signals  in  the  flight  test  telemetry  data  package. 

2.  Seeker  Models 

As  an  adjunct  to  specific  HWIL  simulation  programs,  digital 
seeker  models  may  be  formulated  by  using  open-loop  seeker  characterization 
test  data  obtained  b.y  the  SSC  in  the  RFSS.  Based  upon  program  requirements, 
a  hierarchy  of  models  of  increasing  realism  and  complexity  can  be  developed. 

The  simplest  elements  of  this  hierarchy,  intended  to  model  high  level  functional 
seeker  system  performance,  may  be  required  to  run  in  real-time  as  part  of  the 
HWIL  simulation  of  other  missile  components.  More  detailed  members  of  the 
hierarchy  are  intended  to  accurately  model  seeker  subsystems.  The  most 
realistic  model,  which  generally  requires  significant  computational  resources 
to  execute,  is  the  detailed  subsystem  model  that  is  integrated  into  a  full 
seeker  system  performance  model.  Seeker  characterization  data  obtained  in  the 
RFSS  can  be  used  to  validate  digital  seeker  models  built  from  seeker  design 
criteria,  thus  assuring  that  seeker  hardware  fulfills  its  design  objectives. 
Seeker  characterization  measurements  in  the  RFSS  can  also  be  used  in  conjunc¬ 
tion  with  digital  seeker  models  to  assess  and  exploit  threat  hardware. 

3.  RF  Environmental  Models 


Fliqht.  dynamics  modeling  is  an  established  discipline,  and 
experience  has  shown  that  validated  models  can  be  developed  with  a  high  degree 
of  confidence.  The  problem  of  validating  a  radar  seeker  model  is  avoided  in 


HWIL  simulation  by  using  actual  seeker  hardware.  The  critical  factor,  there¬ 


fore,  in  establishing  the  credibility  of  RFSS  HWIL  simulation  results  lies 


in  the  V&V  of  RF  Environmental  Models. 


The  general  approach  to  environmental  modeling  followed  at  the 
RFSS  involves  forming  hierarchies  of  generic  models  for  each  RF  environment 


element.  These  hierarchies  -  which  exist  for  targets,  clutter,  jet  engine 
modulation,  propeller  and  helicopter  blade  modulation,  multipath  and  chaff  - 
range  from  simple  to  complex,  with  corresponding  ranges  of  applicability  and 
realism.  They  are  generic  in  the  sense  that  their  formulation  is  driven  by 
specific  data  bases  generated  empirically  or  analytically  which  are  appro¬ 
priate  for  both  the  weapon  system  and  the  target  or  threat  vehicle  being 
simulated.  The  target  model  hierarchy,  for  example,  consists  of  four  types: 

o  Isotropic  Scatterer  Model 

o  Empirical  Scatterer  Model 

o  Statistical  Model 

o  Deterministic  Multiple  Scatterer  Model. 

The  simplest,  the  Isotropic  Scatterer  Model,  consists  of  a  point  reflector 
located  in  space  at  the  target  centroid  with  a  fixed  radar  cross  section  (RCS). 
The  Empirical  Scatterer  Model  allows  for  slow  variation  with  aspect  angle  of 
both  the  target  RCS  (amDlitude  scintillation)  and  apparent  angular  position 
(low-frequencv  glint  or  bright-spot-wander).  The  Statistical  Model  adds  to 
the  Empirical  Model  high  frequency  amplitude  scintillation  (rapid  variation 
with  aspect  anale)  and  angular  glint  components  which  may  be  either  aspect 
or  aspect-rate  dependent.  The  final  and  most  realistic  member  of  the  hierarchy, 
the  Deterministic  Multiple  Scatterer  Model,  treats  the  target  as  a  collection 
of  point  scatterers.  Each  scatterer  can  have  aspect-dependent  amplitude  and 
phase  scattering  properties,  with  the  total  target  return  computed  as  the 
coherent  superposition  of  the  returns,  from  the  individual  scatterers  illuminated 
b.y  the  radar  transmitter.  This  results  in  the  seeker  receiving  realistic 
amplitude  scintillation  and  range  and  angle  glint. 

Each  of  these  models  is  driven  by  an  empirical  or  semi -empirical 
data  base,  and  the  extent  to  which  a  particular  model  realistically  represents 
the  radar  signature  of  a  particular  target  or  threat  vehicle  depends  in  large 
measure  upon  the  quality  and  completeness  of  the  available  data  base.  The 
degree  of  realism  required  in  the  simulation  is  also  dependent  upon  the  seeker 
processing  logic.  It  is  the  joint  responsibility  of  the  customer  and  the 
RFSS  Systems  Leader  to  agree  during  the  coordinating  and  planninq  stage  of 
the  simulation  development  cycle  upon  model  specifications  which  both  satisfy 
the  customer's  test  objectives  and  are  imolementable  in  the  RFSS.  The  customer 
is  responsible  for  providing  the  data  from  which  the  models  are  constructed  or 


for  agreeing  to  use  an  existing  RFSS  data  base.  The  RFSS  Systems  Leader  is 
responsible  for  correctly  implementing  these  models;  i.e.,  to  verify  that 
the  RF  signals  presented  to  the  seeker-under- test  meet  previously  agreed-to 
specifications.  The  RFSS  approach  to  Environmental  Model  Verification  is 
discussed  in  detail  in  Section  III. 

The  credibility  of  the  simulation  results— the  degree  of  confidence 
which  one  places  in  these  results— is  largely  dependent  upon  the  realism  of 
the  environmental  models.  Establishing  this  credibility  is  the  purpose  of  the 
model  validation  effort.  One  of  the  first  steps  in  the  validation  process  is 
to  demonstrate  that  the  models  reproduce  the  data  from  which  they  are  derived. 

A  more  comprehensive  validation  procedure  involves  the  quantitative  comparison 
of  model  outputs  with  independent  measurements  or  theoretical  analyses.  It 
is  important  that  the  RFSS  customer  help  to  identify  the  level  of  validation 
appropriate  for  his  specific  program  and  take  the  necessary  steps  t.o  provide 
the  required  validation  data.  The  general  RFSS  approach  to  environmental 
model  validation  is  discussed  in  Section  IV. 

4.  Overall  Simulation 

Verification  of  the  overall  simulation  is  accomplished  by 
demonstrating  the  correct  interface  among  independently  verified  component 
models.  Validation  of  the  overall  simulation  is  also  contingent  upon  the 
validity  of  the  component  models  and  is  accomplished  by  comparing  flight  test 
results  with  simulation  results.  For  example,  the  comparison  of  fin  deflec¬ 
tion  commands  which  served  in  paragraph  B.2  to  validate  the  Flight  Dynamics 
Model  could  also  be  used  to  validate  the  overall  simulation  of  realistic  RF 
Environmental  Models  are  used  in  the  simulation. 

C.  V&V  As  An  Existing  Technical  Discipline 

V&V  techniques  have  been  heavily  used  in  the  software  systems 
environment  for  several  years.  Although  many  such  techniques  have  been  extra¬ 
polated  to  hardware  or  mixed  software/hardware  environments,  there  are 
differences.  In  the  pure  software  situation,  the  actual  item  to  be  evaluated 
is  usually  available  and  can  be  used,  while  in  the  hardware-type  situation 
some  portion  (ranging  from  none  to  all)  of  the  hardware  may  exist  or  may  need 
to  be  simulated.  However,  the  general  software  V&V  techniques  can,  to  some 
degree,  be  transferred  to  the  mixed  software/ hardware  V&V  domain.  They  are 


reviewed  here  as  pertinent  background  to  the  development  of  V&V  methodologies 
specific  to  the  RFSS  and  particularly  the  RF  Environmental  Models. 

1.  General  V&V  Concept 

The  main  purpose  of  V&V  is  to  assure  that  a  developed 
product  satisfies  the  user's  requirements.  To  assure  objectivity,  the  V&V 
team  is  at  times  composed  of  members  who  are  independent  from  the  system 
designers,  developers,  and  implementers.  The  V&V  process,  ideally,  should 
be  accomplished  in  a  series  of  steps  that  interface  with  the  system  develop¬ 
ment.  Several  steps  constitute  a  phase  of  software  system  development,  and 
each  phase  provides  a  definitive,  verified  baseline  for  the  next  phase.  Veri¬ 
fication  is  an  interactive  process  aimed  at  determining  whether  the  product 
of  each  step  and/or  phase  of  the  development  cycle  fulfills  all  the  require¬ 
ments  levied  by  the  previous  step  and/or  phase,  while  validation  is  the  process 
of  testing  the  developed  system  and  comparing  the  results  to  the  required 
performance. 

2.  Software  V&V 

The  described  V&V  process  is  well  suited  for  assuring  that 
developed  computer  software  products  actually  fulfill  the  role  for  which  they 
were  intended.  It  helps  to  prevent  software  costs  from  escalatinq  by  providing 
start-to-finish  traceability.  The  production  and  documentation  of  each  phase 
of  V&V  are  compared  with  those  of  earlier  ones,  and  a  report  of  the  results  is 
prepared.  Figure  3  presents  a  simplified  view  of  such  a  V&V  process.  The 
development  of  a  product  can  begin  at  any  one  of  several  points:  initial  con¬ 
cept  formulation,  research  and  development  design,  operational  production, 
etc.  Sometimes  verification  involves  comparing  several  steps  or  phases 
rather  than  just  two;  the  same  is  true  for  validation. 

Software  verification  involves  reviewing  and  analyzing  all 
system  software-related  deliverables  for  correctness,  completeness,  consis¬ 
tency,  and  pertinence.  The  analyses  are  the  result  of  comparing  the  contents 
of  the  sysfam  deliverables  with  the  requirements,  design  documents,  specifi¬ 
cations,  and  standards  established  by: 

o  Purchase  Description  or  Statement  of  Work 
o  Outline  Acquisition  Plan 
o  Pertinent  Military  Standards  and  Regulations 
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Figure  3.  Simplified  view  of  the  V&V  process, 
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o  Pertinent  Documentation  Standards 
o  Pertinent  Department  of  Defense  Directives 

After  reviewing  and  analyzing  individual  system  documents, 
deficiencies  resulting  from  noncompliance  and  nonconformance  with  the 
established  requirements,  specifications,  and  standards  are  noted;  deficiencies 
are  reported  in  technical  reports  with  changes  recommended  to  correct  the 
deficiency.  Subsequent  versions  of  software-related  documents  are  reviewed 
to  ensure  that  the  customer-di rected  changes  have  been  included. 

Validation  involves  activities  which  ensure  that  the  system 
software  is  adequately  tested  and  meets  established  user  requirements  for 
performance  and  reliability.  Validation  activities  include  independent,  tests, 
system  software  contractor's  tests,  independent  comparative  analyses,  and 
acceptance  testing. 

A  typical  approach  to  a  software  V&V  plan  results  in  a  six 

phase  effort: 

o  System  Design  Verification 
o  Software  Design  Verification 

o  Computer  Program  Configuration  Item  (CPCI)  Verification 

o  CPCI  Validation  and  Integration 
o  Specification  Verification 
o  System  Validation 

For  each  phase  a  chart  similar  to  a  Program  Evaluation  Review  Technique  (PERT) 
chart  is  valuable  to  portray  graphically  all  software-related  V&V  activities, 
time-lines,  deliverables,  milestones,  and  their  inter-relationships  within 
the  program. 

3.  Applicability  of  Software  V&V  Techniques  to  RF 
Environmental  Models 

V&V  for  pure  software  systems  is  more  straightforward  than 
for  the  mixed  hardware/software  configurations  used  in  HWIL  simulation.  These 
configurations  are  not  fixed--they  must  often  be  changed  to  satisfy  specific 
customer  requirements.  The  RFSS  does  not  produce  a  product  in  the  traditional 
software  sense,  so  that  many  of  the  procedures  outlined  are  not  applicable; 
moreover,  certain  items  (Outline  Acquisition  Plan,  Pertinent  Military  Standards 


and  Regulations,  etc.)  are  not  available  to  serve  as  V&V  standards.  '  However, 
the  planned  phased  app*oach  employed  in  software  V&V  is  broadly  applicable 
to  V&V  of  RF  Environmental  Models  and  is  used  in  the  following  sections. 

III.  VERIFICATION  OF  RF  ENVIRONMENTAL  MODELS 

A.  General 

Verification  as  applied  to  RF  Environmental  Models  In  the  RFSS  has 
the  goal  of  assurinq  that  the  electromagnetic  fields  presented  to  the  seeker- 
under-test  during  simulation  meet  the  specific  design  objectives  agreed  to  by 
the  customer  and  RFSS  Systems  Leader  at  the  beginning  of  the  simulation  develop¬ 
ment  cycle.  Validation  addresses  the  broader  issue  of  whether  or  not  these 
correctly  implemented  models  are  an  adequate  representation  of  the  real-world 
RF  environment.  Verification  has,  in  a  sense,  a  binary  output;  if  a  particular 
design  objective  is  not  met  by  verification  testing,  corrective  action  is 
implied.  This  action  can  be  accomplished  either  by  correcting  errors  or 
otherwise  improving  the  model  or  by  relaxing  the  design  specification  if 
demonstrated  to  be  overly  ambitious. 

A  typical  simulation  of  an  RF  guided  missile  in  the  RFSS  is  accom¬ 
plished  in  the  following  phases: 

o  Coordination  and  Planning, 
o  Simulation  Development, 
o  Integration  and  Checkout, 
o  Simulation  Operations, 
o  Analysis  and  Documentation. 

There  are  three  distinct  verification  tasks  which  applv  to  each  RF  environ¬ 
mental  model  and  which  occur  at  different  stages  in  the  simulation  develop¬ 
ment  cycle.  These  are: 

o  Model  Design  Requirement  Review, 

o  Model  Implementation  Verification, 

o  Periodic  Calibration/Diagnostic  Maintenance. 

Model  Design  Requirement  Review  occurs  during  the  Simulation  Development 
phase.  It  consists  of  verifying  that  the  mathematical  models  (typically 
non-real  time  FORTRAN  computer  programs)  developed  for  the  simulation,  satisfy 
the  objectives  set  forth  in  the  coordination  and  planning  phase*  This  task 


15 


is  important  from  a  technical  review  perspective,  particularly  in  its  role 
to  identify  and  isolate  high  risk  areas  which  could  later  impact  schedule 
milestones. 

The  second  task,  Model  Implementation  Verification,  assurs  that 
environmental  models  have  been  correctly  implemented  in  the  RFSS.  This  imple¬ 
mentation  involves  various  processes,  including  microcoding  of  the  mathematical 
model  into  real-time  software,  digital-to-analog  signal  conversion,  in-phase 
and  quadrature  weighting,  time  delays,  amplitude  modulation,  doppler  modula¬ 
tion  and  amplification  and  radiation  of  the  appropriate  range  attenuated  RF 
siqnal  from  the  proper  position  on  the  RFSS  array.  Verification  of  the 
implemented  model  is  accomplished  by  measurement  at  various  stage*  of  signal 
generation.  Final  verification  is  accomplished  by  measurement  of  the  actual 
signals  presented  to  the  seeker-under-test,  utilizing  the  RFSS  Verification 
Receiver  Measurement  System  (VRMS).  Verification  measurements  are  discussed 
further  in  paragraph  III.  D. 

Model  Implementation  Verification  occurs  during  both  the  Simulation 
Development  and  the  Integration  and  Checkout  phases  of  the  simulation.  Real¬ 
time  software  and  signal  generation  through  the  RF  stage  are  verified  during 
Simulation  Development.  By  accomplishing  this  before  missile  hardware  is 
scheduled  to  arrive  in  the  RFSS,  problems  are  detected  and  corrected  with 
minimal  impact  on  schedule  milestones.  This  increases  the  productive  use  of 
missile  hardware  during  the  period  when  it  is  in  the  RFSS,  and  enhances  the 
efficiency  and  effectiveness  of  the  simulation  program.  Verification  of  the 
propagated  RF  signal  using  VRMS  (which  requires  dedication  of  an  aperture  in 
the  RFSS  chamber)  is  generally  performed  during  Integration  and  Checkout. 

The  final  element  of  environmental  model  verification  occurs  on  a 
periodic  basis  during  the  Simulation  Operations  phase  of  the  simulation  cvcle. 
This  task  consists  of  a  variety  of  daily  and/or  weekly  calibration  and  diag¬ 
nostic  maintenance  checks  which  are  designed  to  assure  that  the  environment;1,! 
models  continue  to  meet  their  design  objectives.  As  with  previous  verifica¬ 
tion  tests,  failure  to  meet  a  specification  implies  corrective  action  before 
the  simulation  proceeds.  Near  real-time  checks  of  the  seeker  cutout  are  also 
possible  during  simulation  using  an  HP5451C  Fourier  Processor.  This  capability 
is  designed  both  to  contribute  to  environmental  model  verification  and  to  faci¬ 
litate  on-line  analysis  of  seeker  performance. 
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B.  Model  Configuration  Management 

The  environmental  modeling  objectives  agreed  to  by  the  customer  and 
the  RFSS  Systems  Leader  during  the  Coordination  and  Planning  phase  of  simula¬ 
tion  development,  may  consist  of  both  model  performance  and  model  construction 
specifications.  These  specifications  define  the  type  and  range  of  parameters 
over  which  the  model  is  to  be  exercised.  For  a  target  model,  for  example, 
factors  such  as  type  of  target,  range  of  azimuth  and  elevation  angles,  maximum 
turning  rate,  and  minimum  missile/target  distance  at  which  the  model  must  be 
realistic  might  be  specified.  The  performance  specification  should  define 
how  accurately  each  model  output  variable  can  and  should  be  simulated.  For 
example,  it  might  be  required  that  target  model  radar  cross-section  as  a  func¬ 
tion  of  azimuth  angle  agree  with  a  known  set  of  scattering  measurements  to 
within  3  dBsm.  Specification  of  the  model  construction  sets  forth  the  tech¬ 
nical  approach  defining  how  the  model  is  to  be  formulated. 

Some  environmental  modeling  objectives  are  ambitious  and  may  require 
elaborate  measurement  programs,  advances  in  environmental  modeling  technology, 
or  the  development  of  new  signal  generation  hardware.  Therefore,  it  may  become 
necessary  to  readjust  these  objectives  during  Simulation  Develooment  or  Integ¬ 
ration  and  Checkout.  Model  Configuration  Management,  which  consists  of  docu¬ 
mentation  of  Model  Performance  and  Construction  Specifications,  is  the  responsi¬ 
bility  of  the  RFSS  Systems  Leader,  and  is  essential  to  the  RFSS  V?V  program, 

C.  Implementation  of  RF  Environmental  Models 

A  hierarchy  of  generic  models  for  targets,  clutter  and  blade  modula¬ 
tion  are  available  for  use  in  the  RFSS,  and  are  discussed  in  the  RFSS  User’s 
Guide  [l].  Models  have  also  been  developed  for  multipath  and  ECM,  and  the 
option  exists  to  utilize  actual  jammer  hardware  to  drive  the  RFSS  ECM  and/or 
main  arrays.  As  discussed  in  paragraph  II.B.l,  these  models  are  generic  in 
the  sense  that  they  are  driven  by  customer-supplied  data  bases  which  are 
appropriate  to  the  particular  seeker,  threat  vehicle  and  operational  scenario 
of  interest. 

The  implementation  of  these  models,  as  illustrated  in  Fiqure  4, 
generally  proceeds  in  a  two-stage  process.  A  FORTRAN  computer  program  not 
required  to  execute  in  real-time  is  first  produced;  this  form  of  the  model  is 
shown  to  satisfy  the  Design  Requirement.  The  FORTRAN  program  then  becomes  the 
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standard  against  which  the  real-time  version  of  the  model  is  verified. 


Fiqure  4.  RF  environmental  model  development  and  implementation. 


An  implemented  model  can  be  subdivided  into  three  general  sections: 
software  which  may  reside  on  more  than  one  computer,  IF  signals  which  undergo 
various  analog  processes  in  order  to  generate  a  signal  with  the  proper  doppler, 
waveform  characteristics  and  time  delay,  and  RF  signals  which  are  amplified 
and  transmitted  from  the  RFSS  array  to  the  seeker-under-test.  Verification 
tests  are  designed  to  assure  that  the  proper  model  outputs  are  produced  after 
each  of  these  stages. 

First,  the  digital  output  of  the  real-time  software  is  compared  with 
the  outDut  of  the  FORTRAN  math  model.  Next,  the  analog  output  of  the  amplitude 
modulators  is  measured  to  ensure  that  digital  in-phase  and  quadrature  voltage 
commands  are  correctly  converted  into  analog  signals  and  summed.  Additional 
tests  are  performed  at  each  stage  of  IF  mixing  when  appropriate;  for  example, 
spectra  of  clutter  signals  are  verified  using  a  spectrum  analyzer.  Figure  5 
illustrates  the  results  of  a  typical  clutter  verification  measurement.  Twenty 
time  slices  of  analog  signals  designed  to  simulate  the  clutter  return  detected 
by  a  CW  semi -active  radar  missile  are  measured  at  IF,  transformed  using  a  DFT 
(discrete  Fourier  transform)  spectrum  analyzer,  and  the  spectra  averaged  to 
reduce  random  fluctuations.  The  resultant  averaged  spectrum  is  then  compared 
to  the  intended  spectrum.  The  amplified  RF  signal  is  measured  before  radia¬ 
tion  to  verify  the  correct  input  to  the  array  switching  network.  Finally, 
the  signal  is  propagated  across  the  chamber  and  measured  at  an  aperture  using 
VRMS  to  verify  that  the  correct  signal  is  presented  to  the  seeker-under-test. 
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Figure  5.  Fourier  analysis  of  RFSS  clutter  signal  generated  by  real-time 
software  for  a  time  slice  of  a  simulated  flight. 

Verification  tests  are  designed  in  this  modular  buildup  fashion  to 
■facilitate  the  isolation  of  problem  areas.  Thus  verification  testing  serves 
as  a  diagnostic  tool  and  assists  in  the  process  of  correcting  errors  or 
oversights . 

D.  Verification  of  Propagated  RF  Signals 

The  ultimate  verification  test  of  RFSS  environmental  models  is  the 
measurement  of  electromagnetic  fields  at  the  seeker  aperture  and  the  demonstra¬ 
tion  that  these  signals  meet  the  model  design  requirements.  These  measurements 
are  made  using  the  VRMS.  The  specific  waveform  characteristics  which  are 
measured  are  dependent  upon  the  characteristics  of  the  seeker  whose  performance 
is  being  evaluated.  For  a  semi-active  CW  doppler- tracking  missile,  for  example, 
it  is  sufficient  to  verify  the  power  and  spectrum  of  the  RF  signal  and  the 
apoarent  angular  position  from  which  it  is  being  radiated.  For  pulse-doppler 
missile  with  multiple  range  gates,  it  is  also  necessary  to  verify  proper  time- 
delay  and  phase  coherency  between  the  signals  in  the  range  intervals  being 
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simulated.  Current  and  projected  RFSS  verification  measurement  capabilities 
are  outlined  in  the  following  subsections. 

1.  VRMS-I 

Current  RFSS  verification  measurements  are  accomplished  by 
replacing  the  seeker-under-test  by  one  of  two  instruments,  the  Calibration 
Sensor  (Cal  Sensor)  or  the  Active  Test  System.  The  Cal  Sensor  is  a  four  or 
five  horn  (dual  configuration)  interferometer  which  measures  angular  position 
on  the  RFSS  array  of  a  CW  signal  in  the  2  to  18  GHz  range.  The  receiver  can 
measure  sional  polarization  and  can  measure  angular  position  as  a  function  of 
polarization.  Absolute  accuracy  of  this  device  is  a  function  of  frequency, 
polarization  and  angular  displacement  [3],  The  Active  Test  System  generates 
a  20  W  peak  pulse  modulated  signal  with  variable  Dulse  repetition  interval 
and  duty  factor.  This  signal  is  transmitted  across  the  chamber  where  it  is 
received  by  the  arrav  and  processed  as  in  an  HWIL  simulation.  The  retrans¬ 
mitted  signal  is  received,  range-gated,  and  processed  by  a  heterodyne  receiver. 
The  output  conststs  of  video  and  detected  IF  signals.  Detected  IF  is  used  to 
verify  time-delays  and  the  video  signal  is  passed  to  a  spectrum  analyzer  for 
environmental  Model  spectral  density  verification. 

Current  verification  measurements  are  generally  limited  to  a 
static  target  scenario;  that  is,  neither  of  these  devices  is  capable  of  track¬ 
ing  a  target  flying  an  arbitrary  trajectory  across  the  array.  Verification 
of  RF  environmental  models  in  a  dynamic  situation  is  therefore  done  inferen- 
tially  utilizing  the  seeker-under- test.  After  the  performance  of  the  seeker 
is  established  versus  the  verified  static  environmental  models,  a  number  of 
baseline  dynamic  scenarios  are  run.  If  the  seeker  performs  as  expected 
based  upon  the  static  tests,  the  environmental  models  are  considered  verified 
in  the  dynamic  situation. 

Figures  6  and  7  illustrate  the  use  of  seeker-derived  data  to 
accomDlish  RF  Environmental  Model  development  and  verification.  In  Fiqure  6, 
the  power  spectrum  of  a  time  slice  of  the  monopulse  sum-channel  wideband  video 
siqnal  telemetered  during  a  missile  flight  test  is  presented.  The  analysis 
was  performed  in  the  RFSS  using  an  HP 5451 C  Fourier  Analyzer.  This  data  was 
used  in  the  formulation  of  RF  environmental  models.  In  Figure  7,  the  results 
of  a  similar  analysis  using  data  obtained  during  the  simulated  flight  of  the 
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Figure  6.  Fourier  analysis  of  time  slice  of  wideband  video  signal  from  a 
test  flight. 


Figure  7.  Fourier  analysis  of  time  slice  of  wideband  video  signal  from 
simulated  flight. 


same  seeker  in  the  RFSS  are  presented.  Use  of  the  HP5451C  in  this  manner 
permits  rapid  analysis  of  seeker  performance  during  simulation. 

2.  VRMS-II 

To  upgrade  the  RFSS  verification  measurement  capabilities,  a  new 
receiver  is  planned.  VRMS-II  will  be  capable  of  measuring  polarization, 
doppler,  angular  position  (azimuth  and  elevation)  and  range  or  time-delay. 
Additionally,  VRMS-II  may  be  positioned  in  the  chamber  near  the  seeker  and  be 
capable  of  making  measurements  simultaneously  with  seeker  operation,  lending 
obvious  flexibility  to  both  environmental  model  and  seeker  verification  possi¬ 
bilities.  Because  it  will  not  be  necessary  to  dismount  the  seeker  from  the 
fliqht  table  and  replace  it  with  VRMS,  significant  economies  in  time  are  possible 
in  the  performance  of  both  verification  and  routine  calibration  measurements. 

This  receiver  will  consist  of  an  interferometer  type  multiple 
horn  arrangement  with  a  receiver  and  digitizer  for  each  horn.  Results  are 
recorded  and  processed  off-line  in  a  timely  but  non-real -time  mode,  allowing 
maximum  flexibility.  Processed  data  is  displayed  in  tabular  or  graphical  form, 
with  hardcopy  available,  and  may  be  stored  digitally  for  future  analysis. 

E.  Typical  RF  Environmental  Model  Verification  Plan 

Measurements  designed  to  verify  the  correct  implementation  of  RF 
environmental  models  are  documented  in  the  RFSS  Simulation  Test  Plan  prepared 
during  the  Simulation  Development  Phase.  These  plans  are  ultimately  reduced 
to  step-by-step  detail  in  the  RFSS  Simulation  Test  Procedures.  An  abbreviated 
version  of  typical  test  procedures  utilizing  VRMS-I  to  verify  target  model  RCS 
and  angular  glint  characteristics  is  presented  in  Figure  8. 

IV.  VALIDATION  OF  RF  ENVIRONMENTAL  MODELS 

A.  Rationale 

To  ensure  credibility  of  RFSS  simulation  results  and  to  define 
limitations  and  bounds  on  the  interpretation  of  these  results,  the  validity 
of  the  basic  RF  Environmental  Models  should  be  demonstrated.  Specifically, 
these  models,  utilized  within  their  domains  of  intended  application,  must  be 
shown  to  match  real-world  environments  within  some  acceptable  level  of  accuracy. 
The  intended  application  in  turn  dictates  the  degree  to  which  complexities  and 
subtleties  associated  with  the  physical  processes  being  represented  in  the 
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TYPICAL  1 ARGET  MODEL  VERIFICATION  TEST  PROCEDURES 

o  MOUNT  ACTIVE  TEST  SYSTEM  ON  FLIGHT  TABLE  AND  CALIBRATE. 

O  CALIBRATE  AND  VERIFY  TARGET  MODEL  RCS  PATTERN. 

1.  SELECT  PROPER  PULSE  MODULATOR  PRF  AND  DUTY  FACTOR. 

2.  CENTER  TARGET  IN  REFERENCE  ARRAY  TRIAD  WITH  TARGET  CENTER  OF  GRAVITY  (C.G.)  AT 
REFERENCE  RANGE  <1000  FEET). 

3.  CENTER  ACTIVE  TEST  SYSTEM  RANGE  GATE  ABOUT  TARGET  C.G.  RANGE  BY  MAXIMIZING 
VIDEO  OUTPUT  LEVEL. 

4.  MATCH  RANGE  GATE  WIDTH  TO  TARGET  EXTENT  BY  MAXIMIZING  VIDEO  OUTPUT  LEVEL  AND 
SIGNAL/NOISE  RATIO. 

5.  ADJUST  RANGE  ATTENUATORS  TO  OBTAIN  A  DETECTED  JF  OUTPUT  APPROPRIATE  FOR  THE 
.  TARGET  NOSE-ON  CROSS  SECTION  AND  C.G.  RANGE. 

6.  ROTATE  TARGET  IN  1°  STEPS  THROUGH  RANGE  OF  ASPECT  ANGLE  FROM  0°  TO  180°  IN  THE 
YAW  PLANE. 

7.  RECORD  VIDEO  AND  DETECTED  IF  LEVELS  AT  EACH  STEP.  FOR  STATISTICAL  MODELS.  OBTAIN 
SUFFICIENT  DATA  AT  EACH  STEP  TO  ESTIMATE  MEAN  WITHIN  A  95%  CONFIDENCE  LEVEL. 

8.  REPEAT  FOR  ROTATION  OF  TARGET  IN  PITCH  PLANE. 

9.  COMPARE  CALIBRATED  MEASURED  DATA  WITH  RCS  DATA. 


•  VERIFY  TARGET  MODEL  ANGULAR  GLINT  PATTERN. 

1.  MOUNT  CAL  SENSOR  ON  FLIGHT  TABLE. 

2.  CALIBRATE  CAL  SENSOR  DIRECTION  OF  ARRIVAL  (DOA)  MEASUREMENT  FOR  REFERENCE 
TRIAD.  (SHOULD  BE  THE  SAME  TRIAD  USED  FOR  RCS  AND  TARGET  MODEL  CHECK-OUT). 

3.  CENTER  TARGET  IN  REFERENCE  TRIAD  (NOSE-ON). 

4.  ALIGN  CAL  SENSOR  WITH  TARGET  C.G. 

E.  ROTATE  TARGET  IN  1°  STEPS  THROUGH  RANGE  OF  ASPECT  ANGLE  FROM  0°  TO  180°. 

6.  MEASURE  DOA  OFFSETS  FROM  C.G.  POSITION  AT  EACH  STEP. 

7.  COMPARE  CALIBRATED  MEASURED  DATA  WITH  GLINT  DATA. 


Figure  8.  Typical  target  model  verification  test  procedures. 


model  must  actually  be  present.  Clearly,  a  radar  seeker  designed  to  be  sensi¬ 
tive  to  some  property  of  the  environment  must  be  exposed  to  a  simulated  environ¬ 
ment  which  includes  that  property  if  the  seeker  is  to  be  appropriately  exercised. 
Thus,  a  hierarchy  of  RF  Environmental  Models  emerges  ranging  from  simple  to 
complex  in  their  characterization  of  the  real-world  environment.  Model  valida¬ 
tion  ensures  that  all  "relevant"  properties  of  the  environment  are,  in  fact, 
accounted  for  in  the  model  and  are  accurately  replicated.  The  term  "relevant" 
in  this  context  is  thus  generically  dependent  on  the  characteristics  of  the 
class  of  seekers.  If  the  same  models  are  to  be  used  to  exercise  and  compare 
different  versions  of  a  specific  device  under  test,  the  models  must  incorporate 
properties  which  may  be  required  by  any  of  the  versions.  The  degree  to  which 
such  properties  are  important  can  be  revealed  bv  sensitivity  analyses  accomplished 
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L;i.'ouqh  variation  uf  model  parameters.  RF  Environmental  Model  hierarchies 
are  intentionally  formulated  to  range  from  simple  to  complex  in  order  to 
facilitate  the  identification  of  trends  and  limitations  in  seeker  performance 
due  to  complexities  in  the  RF  environment. 

Quantification  of  the  validity  of  environmental  models  becomes  a 
kev  factor  for  decision-making  in  simulation  planning,  by  ensuring  proper 
application  of  the  models  and  by  establishing  the  level  of  confidence  to  be 
associated  with  simulation  results  based  on  use  of  these  models. 

3.  Data  Requirements 

Validation  procedures  necessarily  are  based  on  comparisons  between 
model  predictions  and  results  of  physical  measurements.  As  illustrated  in 
Figure  9,  data  produced  by  the  physical  measurements  must  be  independent  from 
those  used  in  formulating  the  model  whose  results  are  be  compared  with  the 
data;  otherwise,  only  model  verification  rather  than  va.idation  will  have  been 
effected.  The  model  must,  of  course,  replicate  the  data  used  in  its  formula¬ 
tion  (as  verification  ensures),  but,  more  importantly,  it  must  have  predictive 
capabilities  which  the  validation  process  demonstrates. 


Figure  9.  Validation  of  RF  environmental  models. 
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Validation  of  RF  Environmental  Models  can  involve  both  direct 
measurements  of  the  environment  itself  and  indirect  measurements  of  the  effect 
of  the  environment  on  a  test  seeker  which  incorporates  target  detection,  dis¬ 
crimination  and  location  logics.  The  seeker  can  be  of  generic  design  or  a 
version  of  the  seeker  under  evaluation.  Thus,  a  variety  of  data  acquisition 
and  analysis  activities  can  generate  data  to  support  the  validation  process. 

What  data  is  actually  specified  is  a  function  of  what  can  be  accomplished 
within  reasonable  time,  cost,  and  technical  feasibility  constraints.  Also, 
the  availability  of  useful  pre-existing  data  is  another  major  consideration 
in  determining  what  should  constitute  tne  data  base  for  model  validation. 

In  planning  a  model  validation  effort,  recommending  several  levels 
of  activity  to  acquire  and  process  validation  data  may  be  appropriate  if  addi¬ 
tional  options  become  apparent  to  improve  model  credibility  with  increased 
effort.  Such  options  should  be  identified  along  with  the  tradeoffs  which 
link  the  increased  levels  of  confidence  in  the  model  predictions  and  the  costs 
incurred  in  achieving  these  gains. 

C.  Validation  Procedures 

It  is  clear  that  the  validation  process  is  basically  open-ended-- 
that  tests  with  increasing  levels  of  sophistication  and  expense  can  be  devised 
which  will  improve  model  credibility,  albeit  with  eventual  diminishing  returns. 
Selection  of  specific  procedures  must  be  based  on  reasonable  allocation  of 
effort  against  the  tradeoffs  cited  in  IV. B.  These  procedures  must  be  definitive 
in  terms  of  what  new  data  are  to  be  obtained,  how  these  measurements  are  to 
be  made,  what  instruments  are  to  be  used  in  their  acquisition,  what  data  reduc¬ 
tion  and  analysis  steps  are  required,  and  what  criteria  are  to  be  used  in  the 
validation  assessment.  If  several  independent  data  sets  and  analysis  proce¬ 
dures  are  involved,  an  important  issue  is  how  the  groupings  of  results  are  to 
be  combined  to  arrive  at  a  "bottom-line"  assessment. 

For  example,  measurements  within  the  RFSS  may  be  involved  in  which  a 
simulated  target  is  assigned  a  position  on  the  array,  then  rotated  stepwise  in 
aspect  angle.  A  measurement  receiver  at  the  intended  location  of  the  seeker- 
under-test  in  an  RFSS  aperture  observes  target  position  and  both  RCS  and  glint 
characteristics.  The  measured  properties  of  the  simulated  target  are  then  com¬ 
pared  with  those  of  the  real-world  target  on  which  the  model  was  based.  If 
replication  is  satisfactory,  the  model  implementation  Is  verified  for  the 
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conditions  tested;  if  not,  the  model  is  corrected  and  verification  measure¬ 
ments  repeated.  Next,  seeker  hardware  can  be  mounted  and  angle  errors 
measured  versus  target  orientation  in  open  loop  tests.  If  captive  seeker 
test  data  are  available  against  a  real  target  rotated  in  space,  comparisons 
of  this  test  data  against  open  loop  simulation  runs  can  produce  validation 
results.  More  broadly,  any  experimental  data  which  independently  checks  the 
predictions  of  the  environmental  model  can  be  used  for  validation.  Care  must 
be  taken,  however,  to  ensure  that  the  model  is  exercised  within  its  intended 
range  of  application;  for  example,  far-field  target  scattering  data  cannot  be 
properly  used  to  validate  a  near-field  target  model.  Such  procedures  are 
particularly  relevant  when  sensors  used  in  the  validation  activity  have  identical 
or  similar  characteristics  to  those  of  the  generic  seekers  whicn  the  environmental 
models  are  intended  to  exercise.  This  applies  to  models  covering  target  character¬ 
istics,  clutter,  multipath,  and  ECM  interference.  Failure  to  achieve  complete 
validation  aver  an  initial  range  of  intended  application  may  lead  to  refinement 
of  the  practical  limits  of  model  application. 

In  a  specific  V&V  plan,  the  procedures  have  to  be  definitized  so  that 
steps  covering  data  acquisition,  processing,  analysis  and  validation  assessment 
can  be  scheduled  and  costed  with  responsibilities  assigned.  Implementation  of 
specific  V&V  procedures  may  be  accomplished  partly  by  the  RFSS  and  its  contrac¬ 
tors  on  the  other  hand  and  the  agencies  and  contractors  associated  with  the 
RFSS  customer  on  the  other.  This  is  established  by  determining  how  and  where 
the  procedures  are  best  carried  out  under  the  circumstances.  The  specific 
V&V  baselin:  plan  with  its  possible  options  delineate  the  recommendations  for 
the  V&V  activity. 

D.  Model  Accuracy  Requirements 


In  performing  a  simulation  in  which  random  or  pseudo-random  effects 
occur,  it  is  important  to  realize  that  it  is  not  generally  necessary  to  Drovide 
a  simulation  that  exactly  reproduces  the  behavior  of  the  actual  system.  For 
example,  consider  a  digital  receiver  whose  input  is  a  definite  signal  plus 
random  noise.  In  simulating  the  input  to  the  receiver,  a  random  number  genera¬ 
tor  can  be  used  to  generate  noise,  This  noise  can  be  made  to  have  the  same 
statistical  characteristics  as  the  real  noise,  such  as  mean,  variance,  amplitude 
distribution  function,  and  correlation  function;  but  it  will  not  be  identical 
to  the  real  noise  (of  course,  separate  samples  of  real  noise  will  not  be  ident.al 
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either).  Errors  will  not  occur  on  the  same  bits  in  the  simulation  as  in  real 
hardware,  but  measured  over  a  long  period  of  time,  error  rates  should  be  the 
same.  Thus,  the  noise  model  can  be  made  correct  in  a  statistical  sense,  but 
does  not  yield  identical  results.  Any  comparison  of  the  simulated  and  real 
results  should  account  for  the  expected  differences,  and  comparisons  should 
be  made  using  aporopriate  statistical  parameters. 

By  the  same  token,  the  Deterministic  Multiple  Scatterer  Target  Model 
may  not  correctly  predict  the  exact  azimuth  angle  at  which  a  null  in  the  radar 
return  occurs.  However,  it  should  predict  the  fact  that  nulls  occur,  along 
with  their  frequency,  depth  and  approximate  angular  locations,  and  the  nulls 
should  vary  with  appropriate  parameters,  such  as  wavelength  and  target  orien¬ 
tation  and  range.  In  comparing  the  simulation  with  flight  test  results, 
statistics  derived  from  the  results  are  important  parameters  to  be  compared 
in  addition  to  the  details  of  the  fine  structure  of  the  return  versus  angle 
or  time. 

E.  Quantitative  Methods  for  Model  Validation 

To  quantify  the  results  of  validation  testing,  a  general  procedure 
is  outlined.  The  procedure  focuses  on  the  level  of  agreement  between  model 
predictions  and  observed  real-world  phenomena  within  the  domain  of  intended 
application  of  the  model.  It  also  deals  with  the  certainty  with  which  such 
agreement  levels  have  been  established.  These  factors  combine  to  produce 
probabilities  that  the  model  outputs  are  within  prescribed  limits  set  forth 
as  performance  specifications  and/or  design  objectives. 

Because  the  model  outputs  are  describable  in  terms  of  a  number  of 
parameters,  each  of  which  is  addressed  in  the  specification,  probabilities 
for  each  parameter  are  treated  first,  then  combined  to  arrive  at  an  overall 
probability  that  the  model  produces  a  satisfactory  aggregation  of  realistic 
outputs.  This  probability  serves  as  the  index  of  model  validation  and  is  a 
function  of  the  range  of  input  conditions  for  which  a  single  Index  is  to  apply. 
If  the  model  operates  over  several  regimes  of  input  conditions,  then  a  separate 
validation  index  should  be  derived  for  each  regime.  Thus,  a  target  model,  for 
example,  might  have  a  long-range,  mid-range,  and  short-range  regime—separately 
validated. 
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Before  addressing  details  of  this  methodology,  it  is  noted  that 
model  validity  as  defined  above,  becomes  a  function  of  the  limits  of  accepta¬ 
bility  for  the  various  model  output  parameters  as  reflected  in  the  design 
objective;  this  is  as  it  should  be.  The  limits  of  acceptability  can  be 
relatively  wide  for  less  critical  parameters  and  tighter  for  more  critical 
ones.  In  addition,  because  of  the  statistical  nature  of  some  of  the  measured 
and  predicted  parameters,  a  probabilistic  treatment  is  warranted  and  leads  to 
a  continuum  of  possible  values  for  the  validation  index. 

If  the  various  model  output  parameters  are  more  or  less  independent 
(not  systematically  coupled),  the  probabilities  of  their  falling  within  the 
design  objective  can  be  combined  multiplicatively  to  arrive  at  an  aggregate 
probability.  If  correlated,  steps  can  be  taken  to  form  derived  parameters 
which  are  uncoupled  and  independent.  In  general,  the  model  output  parameters 
to  be  examined  may  be  instantaneous  outputs  or  derived  quantities  such  as 
statistical  means  and  standard  deviations  obtained  from  a  continuous  time 
series. 

More  specifically,  the  procedure  consists  of 

o  Identifying  the  range  of  model  input  conditions  for  which  the 
outputs  are  to  be  compared  with  real-world  observations  or  with 
independent  analysis  results. 

o  Identifying  the  model  outputs--specifically  the  set  of  parameters 
to  be  used  to  characterize  the  outputs. 

o  Establishing  the  limits  of  acceptability  for  each  parameter  in 
terms  of  its  replication  of  real-world  phenomena. 

o  Identifying  the  sets  of  data  to  be  compared. 

o  Performing  comparative  analyses  and  deriving  key  statistics 
including  means  and  standard  deviations  for  each  set  to  be 
compared.  Por  simulation  runs,  input  conditions  over  the 
applicable  range  are  varied  to  develop  the  requisite  distribution 
of  outputs. 
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o  Deriving  measures  of  closeness  of  the  parameters  being  compared 
(based  on  differences  of  the  means)  and  certa.  ity  of  the  assess¬ 
ment  (based  on  analysis  of  confidence  limits  as  affected  by  the 
sparsity  of  the  data). 

o  Combining  these  two  measures  to  compute  the  probability  that  the 
agreement  is  within  specified  tolerance. 

o  Further  combining  such  assessments  as  might  be  obtained  from 
independent  test  series  dealing  with  the  same  parameter,  using 
a  linear  combinatorial  scoring  procedure  in  which  weighting 
factors  account  for  the  adjudged  relevance,  comprehensiveness 
and  reliability  among  the  several  tests.  The  result  here  is  a 
best  overall  estimate  of  the  probability  that  the  model  output 
parameter  being  examined  is  within  tolerance. 

o  Combining  probabilities  for  all  the  parameters  characterizing 
model  outputs.  Individual  probabilities  are  multiplied  for 
independent  parameters.  Correlated  parameters  are  to  be  uncoupled 
by  additional  procedures. 

o  Performing  sensitivity  analyses  by  varying  the  limits  of 
acceptability  in  the  model  specifications  and  checking  the 
degree  to  which  validation  indexes  change. 

o  Identifying  potential  problem  areas  in  model  assumptions  and 
implementation  factors;  institute  trial  changes  in  model  formu¬ 
lation  to  assess  corresponding  effects  in  model  performance 
and  recalculating  validation  indexes. 

Figure  10  summarizes  the  key  procedural  steps  involved  in  comparing 
model  outputs  with  corresponding  observation  of  real-world  phenomena  to  arrive 
at  an  overall  validation  index.  The  sensitivity  of  the  index  to  model  assump¬ 
tions,  model  inputs  and  specified  tolerance  on  model /real -world  agreement  is 
also  determined  as  part  of  the  procedure  to  fine-tune  the  model  performance 
requirements.  The  procedure  is  susceptible  to  iteration  during  weapon  system 
development  as  new  simulation  and  flight  test  results  are  made  available.  Such 
iterations  tend  to  reduce  risk  and  improve  confidence  in  interpreting  simula¬ 
tion  results. 
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The  analytical  techniques  and  procedures  for  accompli shinq  the 
above  steps  are  available  to  be  utilized  as  applicable.  It  is  recognized 
that  the  degree  to  which  deployment  of  these  procedures  is  warranted  is  highly 
dependent  on  the  nature  of  the  specific  program  and  the  availability  of  data 
for  comparative  analysis.  The  general  methodoloa.v  thus  serves  as  a  framework 
for  subsequently  defining  procedural  steps  in  quantifying  validation  results 
in  specific  programs. 

One  of  the  principal  techniques  utilized  deals  with  comparisons  of 
two  finite-sample  distributions  representing,  for  example,  a  model -produced 
parameter  and  a  corresponding  observed  real-world  parameter.  The  technique 
involves  computation  of  the  finite-sample  means  and  standard  deviations;  the 
differences  of  the  two  means  serve  as  a  best  estimate  of  the  agreement  between 
the  model  and  the  real  world  with  respect  to  that  parameter.  The  well-known 
"t  statistic"  is  utilized  in  an  analysis  which  yields  the  probability  that  the 
true  difference  in  means  lies  within  a  specified  interval  about  zero  differ¬ 
ence  (Appendix  A. outlines  the  basic  procedure).  This  probability  becomes  the 
validation  index  for  that  parameter  with  respect  to  the  allowed  tolerance  on 
model  realism. 

Appendix  B  outlines  other  techniques  and  procedures  that  have  been 
studied  for  use  in  quantifying  validation  results.  All  relate  to  the  compari¬ 
son  of  measured  data  with  simulation  results  obtained  by  operating  directly  on 
the  data  to  be  compared  or  on  parameters  derived  therefrom.  If  the  observed 
data  is  of  the  nature  of  a  time  series,  there  needs  to  occur  a  combining  over 
the  samole  elements  of  the  series  in  the  comparison.  The  usl-  of  Thiel's 
Inequality  Coefficients  (Section  8-1)  and  the  Cross  Correlation  Coefficient 
(Section  B-2)  are  two  aporoaches  to  accomplish  this.  If  the  parameters  to  be 
compared  are  random  variables,  comparisons  frequently  are  affected  by  data 
sparsity  or  by  measurement  errors;  thus,  the  inclusion  of  measures  of  certainty 
of  assessment  in  the  procedure  outlined  in  Figure  10.  One  possible  technique 
to  handle  sparsity  of  test  data  in  such  cases  is  to  utilize  Bayesian  Updating 
(Section  B-3)  to  produce  the  probabilities  required  by  means  of  hypothesis 
testing.  Finally,  in  combining  the  results  of  several  diverse  comparisons 
dealing  with  the  same  physical  parameters,  appropriate  combinatorial  techniques 
are  needed.  Section  B-4  outlines  a  simple  linear  scoring  procedure  involving 
weighting  factors  which  can  account  for  the  relevance,  comprehensiveness  and 


reliability  of  the  various  independent  test  results  used  in  comparisons 
leading  to  a  validation  index. 

F.  Specific  RF  Environmental  Model  Validation  Plan 

Where  validation  of  one  or  more  of  the  RF  Environmental  Models  is 
required  in  an  overall  simulation  program,  a  specific  validation  plan  is  for¬ 
mulated  for  an  effort  involving  participation  by  both  the  RFSS  and  the  customer. 
The  initial  plan,  including  options  as  appropriate,  is  drafted  by  the  RFSS  and 
reviewed  by  the  customer.  The  approved  plan  reflects  the  final  selection  made 
among  validation  testing  options,  delineates  schedules  and  costs,  identifies 
the  joint  responsibilities  of  the  RFSS,  the  customer,  and  supporting  organi¬ 
zations,  and  should  relate  to  the  validation  of  the  overall  simulation. 

The  actual  data  to  be  utilized  and  comparisons  to  be  made  are  highly 
program-specific;  these  depend  on  the  availability  and/or  feasibility  of 
acquiring  various  lab  and  field  test  data,  computer  simulation  outputs  and 
flight  test  results.  The  analytical  procedures  to  be  used  in  the  quantitative 
evaluations  are  drawn  from  those  delineated  in  paragraph  IV. E,  as  applicable, 
and  supplemented  where  necessary  with  other  specialized  techniques.  The 
degree  of  specificity  needed  is  comparable  to  that  in  the  examole  of 
paragraph  III.E. 

The  formulation  of  validation  testing  options  in  this  draft  plan 
implies  a  cost/benefit  tradeoff  in  obtaining  higher  levels  of  confidence  in 
the  models  at  additional  expense  in  time  and  dollars.  The  plan  sets  forth 
the  tradeoff--making  estimates  of  confidence  level  improvements  based  on 
analysis  of  the  volume  and  quality  of  the  data  to  be  compared  and  the  power 
of  the  data  reduction  methods  to  be  used. 

G.  Relationship  to  Weapon  System  Validation 

Evaluation  and  validation  of  weapon  system  performance  is  a  cumu¬ 
lative,  hierarchial  process.  The  standard  weapon  system  development  cycle, 
as  illustrated  in  Figure  11,  is  characterized  by  four  phases  and  three  primary 
program  milestones  at  which  Defense  Systems  Acquisition  Review  Council  (DSARC) 
reviews  are  held  and  program  continuation  decisions  made.  The  capability 
of  the  hardware  to  fulfill  technical  and  operational  reauirements  must  be 
evaluated  at  each  decision  point.  Simulation,  both  all-digital  and  HWIL, 
provides  major  inputs  to  this  decision  process. 
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Figure  11.  Weapon  System  Development  Process, 

During  the  conceptual  phase,  when  a  system  design  is  formulated  to 
meet  both  technical  arid  operational  requirements,  simulation,  particularly 
all-digital,  is  a  valuable  tool  in  performing  parametric  system  analyses.  HWIL 
simulation  is  sometimes  used  during  the  validation  phase  to  demonstrate  that 
brassboard  hardware  meets  its  design  requirements,  and  also  to  improve  system 
performance.  In  combination  with  flight  tests,  simulation  is  an  integral 
factor  in  determining  the  performance  of  the  fully-developed  system,  and  can 
make  important  contributions  to  the  final  DSARC  decision  to  enter  production. 
Finally,  after  the  system  is  produced  and  deployed,  simulation  provides  a 
timely,  cost-effective  and,  if  security  requirements  dictate,  covert  method 
of  evaluating  and  optimizing  system  performance  in  new  scenarios  and  against 
changing  threat,  capabilities. 

To  contribute  to  the  decision-making  process,  the  results  and  predic¬ 
tions  of  simulation  must  be  realistic  and  credible.  RF  Environmental  Model 
verification  and  validation  form  part  of  the  groundwork  for  a  validated  simu¬ 
lation,  wh  ch  in  turn  contributes  to  weapon  system  validation.  Thus,  a  vali¬ 
dation  hierarchy  can  be  constructed,  with  the  validity  of  the  top  levels 
depending  critically  on  the  validity  of  each  lower  level. 

The  validation  process,  as  illustrated  in  Figure  12,  can  be  viewed 
as  building  a  pyramid  of  confidence  in  weapon-system  performance  predictions. 

As  new  scenarios  are  introduced,  sensitivity  analyses  performed,  models 
improved,  and  simulation  predictions  corroborated  with  flight,  test  data  and 
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other  independent  analysis  results,  the  knowledge  base  of  the  pyramid  is 
broadened  step  by  step  and  higher  levels  of  confidence  reached  over  a  period 
of  time.  With  simulation  results  supported  by  a  carefully  structured  V&V 
foundation,  technical  and  program  managers  are  able  to  make  difficult  weapon 
system  development  decisions  with  increased  confidence  and  decreased  risk. 
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(REALISM, 
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SIMULATION 


SENSITIVITY  ANALYSES 
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Figure  12.  Validation  -  building  a  pyramid  of  confidence. 


Validation  cannot  be  achieved  through  a  single  experiment  or  flight 
test.  Each  input  to  the  validation  process  generates  new  insights  into  system 
performance.  As  deficiencies  are  corrected  and  results  begin  to  corroborate 
and  support  each  other,  confidence  is  developed  about  predicting  performance. 
100%  confidence  can  never  be  reached  because  not  all  possible  scenarios  and 
contingencies  can  be  explored  in  validation  testing,  but  an  iterative  valida¬ 
tion  program  carried  opt  over  a  period  of  time  can  be  made  to  reduce  risk  and 
uncertainty  to  acceptable  levels. 


V. 


SUMMARY 


RF  Environmental  Models  utilized  in  HWIL  simulation  of  radar  guided 
missiles  at  the  MICOM  ASC  are  developed  in  hierarchies  ranging  from  simple 
to  complex.  These  varying  degrees  of  realism  permit  the  parametric  assess¬ 
ment  of  seeker  sensitivity  to  features  of  the  RF  environment  and  facilitate 
the  identification  of  trends  and  limitations  in  seeker  performance.  The  RFSS 
approach  to  verification  and  validation  of  these  models  is  based  on  the  follow 
ing  multi-phase  program,  each  phase  building  upon  the  results  of  the  previous 
one: 

o  Verification  of  RF  Environmental  Model  design. 

o  Verification  of  real-time  hardware/software  model  implementation 

o  Routine  calibration/diagnostic  maintenance  to  ensure  continued 
correct  implementation  during  simulation  operation. 

o  Model  validation  using  independent  corroborative  data  and 
quantification  of  the  degree  of  model  realism. 

Implementation  of  this  program  involves  joint  participation  by  the  RFSS 
staff,  the  RFSS  customer  and  various  supporting  organizations—in  accordance 
with  an  approved  V&V  plan.  Key  activities  include  identifying,  provisioning 
and  analyzing  data  required  for  model  validation.  The  V&V  proqram  provides 
the  means  for  assessing  the  credibility  of  simulation  results,  and  lays  the 
foundation  for  properly  interpreting  simulation  results  as  required  for 
program  planning  and  decision-making  in  weapon  system  development. 
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APPENDIX  A 


CONFIDENCE  INTERVAL  FOR  DIFFERENCE  IN  MEANS 
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Assuming  that  samples  are  drawn  from  two  populations,  P  and  A,  and 
that  these  populations  are  normally  distributed  and  have  equal  variance,  the 
following  procedure  can  be  used  to  determine  the  confidence  that  the  true 
difference  between  the  means  does  not  exceed  some 'specified  tolerance  value. 

The  statistic 

t  =  [ (P-A)  -  (Pp  -  pA)]*J 


has  the  t-distribution  with  m  =  np  +  nA  -  2  degrees  of  freedom,  where 

nPnA'nP  +  r,A"  ^ 

(np  +  n^)  (npSp  +  n^S^) 
and  F  and  A  are  the  means  of  the  samples.  Sp  and  are  the  standard  deriva¬ 

tions  of  the  samples,  n  and  nA  are  the  wes  of  the  samples  (number  of  points 
measured)  and  up  and  are  the  means  (unknown)  of  the  populations. 

To  find  the  100  (1-0%  confidence  interval,  enter  the  t-distribution 
table  with  e  and-  m  degrees  of  freedom  to  find  a  value  of  t.  Using  this  t, 
the  confidence  limits  (symmetrical  about  P-A)  are  simply 

(P-A)  ±  -J 

For  50%  confidence,  the  value  t/J  is  the  probable  error. 


On  the  other  hand,  there  may  be  predetermined  tolerance  limits,  L\  and 
L2,  within  which  it  is  required  that  the  true  value,  up  -  uA,  lies.  The 
tolerance  limits  may  or  may  not  be  symmetrical  about  (F-A).  For  each  limit, 


L. ,  calculate 

L' 


ti  =  |  (P-A)  -  l^l-O 


Using  this  value  of  and  m  degrees  of  freedom,  enter  the  t-distribution 
table  and  find  by  interpolation  the  corresponding  value  of  ef .  For  most 
tables,  t  represents  the  probability  measure  of  both  tails  of  the  t-distri¬ 
bution,  so  each  cj  must  be  halved  and  then  added  together.  Thus,  the 


confidence  becomes 


that  the  interval  defined  by  the  limits  L:  and  Lz  contains  the  true  value  of 
the  difference  between  the  means,  (up  -  u^). 
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This  approach  has  the  advantage  that  it  works  with  small  sample  sizes, 
although  at  least  one  sample  must  be  of  at  least  size  two  or  more.  It  requires 
the  assumption  of  normality,  but  is  reasonably  accurate  for  distributions  that 
differ  somewhat  from  normal.  Finally,  it  assumes  that  the  variance  of  both 
populations  are  the  same,  but  the  results  do  not  appear  to  be  sensitive  to 
this  assumption,  particularly  when  the  two  sample  sizes  are  similar.  This 
latter  assumption  can  be  tested  using  the  F-distribution,  but  that  test 
requires  at  least  two  sample  points  in  both  samples. 


APPENDIX  B 


QUANTITATIVE  TECHNIQUES  APPLICABLE  TO  VALIDATION  TESTING 


In  this  Appendix  several  techniques  are  presented  that  are  applicable 
in  determining  the  degree  to  which  the  simulation  agrees  with  measured  per¬ 
formance. 


8-1  THIEL 1 s  INEQUALITY  COEFFICIENTS 


Thiel's  Inequality  Coefficients  (TIC)  are  used  to  compare  two  sampled 
time  series.  The  predicted  time  series  P  is  intended  to  represent  the  actual 
time  series  A.  The  basic  coefficient  u  is  given  by: 


r  i=l 


iTTT-tTiT 

1-1  i=l 


where  P-  and  A.^  are  the  sampled  values  of  the  predicted  and  actual  time 
series  respectively,  and  n  is  the  number  of  samples  of  each.  U  varies  from 
zero  to  one,  with  zero  corresponding  to  identical  series  and  one  correspond¬ 
ing  to  very  dissimilar  series. 


V  can  be  decomposed  into  three  components: 

P-A 

U  =  -0—  unequal  central  tendency 

VSA 

Us  =  — 0 —  unequal  variation 


uca 


VZO-rlSS 

- — r —  imperfect  covariation 


where  n  is  the  denominator  of  Equation  1,  P  and  A  are  the  means  of  and  Aj, 
r  is  the  correlation  coefficient  between  and  H.  Note  that 


U  a 
m 


+  Us* 


uc2  -  UJ 
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Equation  B2  may  be  rewritten  as: 
Um  +  IIs  +  Uc  =  1 


(B3) 


where 


um  =  ( 

k  u  / 

bias  proportion 

us  .1 

N1 

\  u  / 

variance  proportion 

f  u.  y 

Uc  =  ( 

iir) 

covariance  proportion 

The  TIC  is  easily  extended  to  the  case  where  there  is  more  than 
one  time  series  associated  with  the  predicted  and  actual  cases. 


B-2  CROSS  CORRELATION  COEFFICIENT 


The  Cross  Correlation  Coefficient  is  a  measure  of  the  similarity  of 
two  sampled  time  series.  It  is  given  by: 

(P..P)  (A. -A) 


R 


I  E 


M 


SPSA 


where  P.  and  A-  are  the  nredicted  and  actual  time  series,  P  and  A  are  the 

'Z'  x- 

means  of  the  PP  s  and  /1^‘s,  Sp  and  S/[  are  the  standard  deviations  and  n  is 
the  number  of  samples  of  each  series. 


R  varies  from  -1  to  +1,  being  +1  if  the  series  Pi  and  ^  are  identical, 
0  if  there  is  no  correlation  between  the  series,  and  negative  if  the  series 
tends  to  be  out  of  phase  with  each  other. 


B~3  BAYESIAN  UPDATING 

The  correlation  of  simulation  results  and  test  data  is  frequently 
hampered  by  the  sparsity  of  the  test  results  in  comparison  with  the  abundance 
of  available  simulation  outputs.  A  technique  Is  needed  for  quantitative 
evaluation  of  the  degree  to  which  the  model  of  the  system  predicts  the  per¬ 
formance  of  the  real  system  as  evidenced  by  actual  test  results.  One  possi¬ 
bility  lies  in  the  use  of  Bayesian  analysis  in  which  test  data  can  be  examined 
in  terms  of  the  probabilities  that  they  derive  from  one  of  several  hypothesized 
model  formulations. 
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In  this  technique,  a  probability  distribution  is  assumed  for  some 
problem  parameter,  only  one  unknown  value  of  which  represents  the  "true  state" 
of  the  system  and  its  environment.  A  particular  value  of  the  parameter  in 
effect  selects  a  hypothesis  concerning  the  real-world  state.  The  distribution 
represents  a  best  estimate  of  the  probability  that  a  given  value  of  the  para¬ 
meter  will  be  the  correct  one,  prior  to  any  observations.  When  observations 
are  made— however  indirect  or  incomplete— the  probability  distribution  becomes 
modified  using  a  specific  updating  procedure.  This  posterior  distribution  is 
again  susceptible  to  further  updating  if  additional  independent  measurements 
pertaining  to  the  same  "true  state"  are  made. 

The  quantity  described  by  the  prior  and  posterior  distribution  may  itself 
be  a  parameter  of  another  statistical  representation.  For  example,  if  miss- 
distance  were  described  by  a  one-parameter  distribution  (such  as  a  Rayleigh), 
then  the  parameter  expressed  in  terms  of  average  miss-distance  is  a  function 
of  the  various  problem  assumptions;  any  one  set  leads  to  a  single  value  of 
the  average  miss-distance  parameter.  Stochastic  processes  consistent  with 
that  set  of  assumptions  in  turn  account  for  the  distribution  of  miss-distances 
about  this  average  value.  Furthermore,  a  number  of  sets  of  assumptions  can 
be  treated  during  simulation,  each  one  associated  with  its  own  miss-distance 
distribution  as  derived  from  simulation  results.  The  problem  then  becomes 
one  of  hypothesis  testing,  specifically  to  determine  which  set  of  assumptions 
is  more  likely  to  be  correct. 

In  the  Bayesian  updating  process,  the  prior  distribution  relates  to 
the  probability  of  correct  choice  among  the  alternative  sets  of  assumptions, 
with  the  observations  tending  to  support  or  refute  the  candidacy  of  any  given 
set.  Of  course,  denser  and  higher  quality  data  are  more  likely  to  sharpen 
the  selection.  However,  the  procedure  does  as  much  as  is  possible  consistent 
with  the  available  data;  it  is  quantitative  and  permits  sensitivity  analysis 
with  the  respect  to  the  assumptions.  As  more  and  more  data  is  available 
regarding  the  "true  state,"  the  initial  assignment  of  the  prior  distribution 
among  these  alternatives  becomes  less  and  less  important  and  the  final  posterior 
distribution  becomes  increasingly  independent  of  the  initial  assignment. 

To  be  more  specific,  consider  N  hypotheses  of  a  miss-distance  distribu¬ 
tion  each  of  which  by  illustration  could  be  characterized  by  a  single  distri¬ 
bution  parameter  y./.  p(m  •)  is  the  initial  assignment  of  the  probability  that 

v  J 
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the  „7^  state  is  in  fact  the  correct  one,  in  other  words  that  u.  is  the 

J 

correct  value  of  average  miss-distance. 


ProUubitity 

P( 


With  the  set  of  assumptions  being  associated  with  y  the  miss- 

«7 

distance  distributions  corresponding  to  several  values  of  j  are  derived  from 
analysis  and  simulation  as  depicted  below. 


A  limited  series  of  measurements  which  correspond  to  a  single  "true 
state"  is  associated  with  a  set  of  observations  "0"  of  miss-distance.  To 
determine  which  value  of  the  index  j  best  fits  the  observations,  the  Bayesian 
formulation  is  utilized  in  which 


where 


P(Uj|0) 


P(Oivij) 

The  quantity  p(o\vj)  can  be  calculated  directly  by  first  computing  for 
each  observation  the  probability  that  a  given  distribution  would  have  produced 
the  measurement,  then  aggregating  the  observations  using  joint  probabilities. 
This  can  be  done  regardless  of  the  sparsity  of  the  data..  That  is,  the  metho¬ 
dology  does  not  depend  on  statistics  being  derived  directly  from  the  observa- 
tions--we  deal  only  with  the  likelihood  that  the  observed  data  came  from  one 

assumed  distribution  or  another.  In  fact,  the  quantity  p(o|u  .)  for  the 
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various  j  which  yields  the  smallest  value  of  p(o |uj)  then  compare  the  remaining 
P's  in  the  form  of  ratios,  specifically  called  likelihood  ratios.  Likelihood 
ratios  thus  are  a  measure  of  the  relative  correlation  of  the  measured  data 
and  the  simulation  results. 

In  short,  Bayesian  updating  can  provide  a  practical  method  for  quanti¬ 
tative  validation  of  simulation  runs  with  sparse  experimental  data. 

B-4  COMBINING  OF  VALIDATION  SCORES  OBTAINED  FROM  DIVERSE  COMPARISONS 

If  several  diverse  comparison  tests  are  conducted  to  check  the  ability 
of  a  model  to  produce  realistic  outputs,  the  individual  validation  scores  can 
be  combined  using  weighting  factors  accounting  for  the  relevance,  comprehensive¬ 
ness  and  reliability  of  the  individual  data  sets  being  compared.  Combinations 
of  weighted  scores  can  be  used  to  produce  higher  level  scores  which  in  turn 
may  thus  be  further  combined  in  a  multi-level  scoring  methodology.  A  combina¬ 
torial  approach  permits  logical  grouping  and  treatment  of  as  many  parameters 
or  factors  as  are  judged  relevant.  Combining  of  scores  can  occur  within  a 
single  level  or  over  several  levels. 


is  the  prior  assignment  of  probability  that  the 

4.U 

ju  hypothesis  is  true 


[2x^)0]. 


*  th 

is  the  posterior  probability  that  the  j  hypothesis 

is  true,  given  the  set  of  observations  "0" 


is  the  probability  that  the  set  of  observations  "0" 
occurs,  given  the  j^  hypothesis. 


In  a  linear  combinatorial  scheme,  for  example,  all  scores  can  be 
placed  on  a  scale  0  to  100  and  weighting  factors  on  a  scale  0  to  1.  By  so 
constraining  these  factors,  the  derived  score  also  has  values  between  0  and 
100  and  is  therefore  in  proper  form  for  use  in  higher  level  combinations.  A 
linear  combinatorial  scheme  is  most  justified  when  the  contributing  factors 
are  independent,  but  the  scheme  can  provide  useful  results  if  dependencies 
are  present  but  are  unknown. 


For  a  two- level  scheme,  a  final  score  S  is  given  in  terms  of  first 
leve'  : cores  and  weighting  factors  w ^  by 

subject  to  the  constraint 


1 

Similarly,  S.  is  given  in  terms  of  second-level  raw  scores  S .  .  and 

weighting  factors  w..  by 

1*3 
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and 
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This  can  be  extended  to  as  many  levels  as  needed.  The  0-to-100  scale 
chosen  for  each  of  the  scores  represents  a  progression  with  increasing  values 
designating  more  favorable  situations.  The  scale  would  represent,  for  instance, 
the  estimated  probability  in  percent  that  a  model  output  is  within  prescribed 
tolerance.  Also,  various  nonlinearities  can  be  introduced  such  as  setting 
unacceptable  raw  scores  (below  a  pre-set  limit)  to  zero  for  the  next  level  of 
combination  in  the  progression  leading  to  a  final  overall  validation  score. 
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